System and method for converting a two-dimensional floor plan into an interactable three-dimensional simulation

ABSTRACT

A system and method for indoor spatial mapping. The system includes a mobile mapping device having a camera; a plan database configured to store 2-D floor plans; and a processor configured to integrate a selected floor plan from the plan database with object data received from the mobile mapping device to create an interactable 3-D spatial mapping of an interior portion of a structure so that one or more objects are moveable relative to a surrounding environment within the 3-D spatial mapping. The processor is configured to recognise objects from an image from the camera and anchor the recognised object within the interactable 3-D spatial mapping.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the U.S. National Stage of International PatentApplication No. PCT/AU2020/050098, filed on Feb. 6, 2020, which claimsthe benefit of and priority to Australia Patent Application No.2019900374, filed on Feb. 6, 2019, the contents of each of which arehereby incorporated by reference in their entireties.

FIELD OF THE INVENTION

The present disclosure relates to improvements in systems and methodsfor rendering a two-dimensional floor plan in a spatial,three-dimensional format.

BACKGROUND OF THE INVENTION

Most buildings do not have a customised or site-specific evacuationtraining system. For those few that have an evacuation training system,such systems rely on two-dimensional (2-D) plans as part of any trainingexercise. Reading and attempting to memorise details from 2-D plans istedious, and many details are not memorised by key individuals.Moreover, 2-D plans are not conducive to use during realistic emergencysituations. Accordingly, there exists a need to provide an improvedsystem and method which is less prone to the problems of conventionalsystems and methods, and provides a more engaging, memorable and safeexperience.

For systems which do render a 3-D plan, such systems fall short sincemany situations and objects that may be encountered and likely to beused are not useable within a simulation.

SUMMARY

The present invention in one preferred aspect provides for a system forindoor spatial mapping. The system includes a mobile mapping devicehaving a camera, a plan database configured to store 2-D floor plans,and a processor configured to integrate a selected floor plan from theplan database with object data received from the mobile mapping deviceto create an interactable three-dimensional (3-D) spatial mapping of aninterior portion of a structure so that one or more objects are moveablerelative to a surrounding environment within the 3-D spatial mapping,the processor being configured to digitally recognise objects from animage from the camera and anchor the recognised object within theinteractable 3-D spatial mapping. Preferably, structural features suchas walls and doorways are interactable as well as objects such as hosereels, fire extinguishers and even lighting.

In another preferred aspect, the present invention provides a method forgenerating an indoor spatial mapping. The method includes storing a 2-Dfloor plan in a floor plan database; digitally tagging reference pointswhich include at least one item of emergency equipment in a level of astructure; anchoring the at least one tagged item of emergency equipmentto the 2-D floor plan; and rendering a 3-D floor plan with at least oneanchored item of emergency equipment, the anchored emergency equipmentbeing interactable within the 3-D floor plan.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed. In the presentspecification and claims, the word “comprising” and its derivativesincluding “comprises” and “comprise” include each of the statedintegers, but does not exclude the inclusion of one or more furtherintegers. It will be appreciated that reference herein to “preferred” or“preferably” is intended as exemplary only.

As used herein, “interaction” or “interactable” is intended to refer toan object or item which is moveable within the simulated environment,compared to surrounding objects, features or items. For example, a usermay interact with an item such as a fire extinguisher by picking it upor handling it, aiming a nozzle or hose, and triggering a release toexpel contents towards a hazard such as a fire. It will be appreciatedthat both objects and positioning within the spatial mapping may beinteracted with to simulate an environment and provide a more realisticsituation. For example, smoky conditions may be included to obfuscatevision when traversing a passage or stairwell. A moving surroundingenvironment may be included to simulate an explosion, or earthquake.Simulated movement(s) or condition(s) may be derived from a databasestoring such data for a particular object or condition preconfiguredprior to inclusion in any simulation.

As used herein, “configured” includes creating, changing, or modifying aprogram on a computer or network of computers so that the computer ornetwork of computers behave according to a set of instructions. Theprogramming to accomplish the various embodiments described herein willbe apparent to a person of ordinary skill in the art after reviewing thepresent specification, and for simplicity, is not detailed herein. Theprogramming may be stored on a computer readable medium, such as, butnot limited to, a non-transitory computer readable storage medium (forexample, hard disk, RAM, ROM, CD-ROM, USB memory stick, or otherphysical device), and/or the Cloud.

The claims as filed and attached with this specification are herebyincorporated by reference into the text of the present description.

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and together with the description, serve to explain theprinciples of the invention.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a system diagram of the components of the system in accordancewith a preferred embodiment of the present invention.

FIG. 2 is an evacuation sign useable with the system of FIG. 1.

FIG. 3 is a perspective view of a floor plan useable with the system ofFIG. 1.

FIG. 4 is a view of a 3-D spatial mapping as seen from a mixed-realityviewing device.

FIG. 5 is a flow diagram of a preferred method of rendering an indoorspatial mapping.

DETAILED DESCRIPTION OF THE DRAWINGS

Reference will now be made in detail to the present preferredembodiments of the invention, examples of which are illustrated in theaccompanying drawings.

FIG. 1 shows a preferred embodiment of a system 100 for indoor spatialmapping having a mobile mapping device 102, a floor plan database 104, acomputer processor 106, and a mixed-reality system 108. In use, a usersurveys the interior of a floor of a building with mapping device 102.Data from the survey is uploaded and stored to database 104. Objects aretagged and anchored within the floor plan using processor 106, andstored as needed in database 104 as an enhanced or latent 3-D floorplan. The latent 3-D floor plan is downloaded from database 104 to amixed-reality system 108 to render an active, explorable indoor 3-Dspatial mapping with objects previously tagged and anchored. Thepreferred elements of system 100 and their interrelationship aredescribed below.

Referring to FIG. 1, mobile mapping device 102 preferably includes auser interface 110, display 112, a camera, a depth detector, and awireless transceiver radio. The depth detector may include a depthdetection laser emitter. The details of depth detection would beappreciated by a person of ordinary skill in the art, and are omittedfor simplicity. The wireless radio is preferably configured for Wi-Fiand/or peer-to-peer communications, such as Bluetooth, Wi-Fi Direct,and/or NFC. It will be appreciated that mapping device 102 may beconfigured for wired communication in addition to, or as an alternativeto wireless communication.

In one or more embodiments, mapping device 102 may be a customised,purpose-build device for indoor spatial mapping, or a speciallyprogrammed smartphone, tablet, or other mobile computing deviceconfigured as desired to perform the spatial mapping functions describedin one or more exemplary embodiments herein.

Floor plan database 104 is configured to store a plurality of digitisedfloor plans. Database 104 may be configured as a physical server, acloud server, a part of a distributed server network, and/or a portablememory, with or without wireless communications.

Processor 106 may be in the form of a microcomputer such as a laptopcomputer. Processor 106 preferably includes a microchip, such as aSystem on Chip (SoC), with appropriate control circuitry.

Mixed reality system 108 is configured to combine features fromreal-world surroundings with features from database 104. In onepreferred embodiment, mixed reality system 108 is configured as avirtual reality system. Mixed reality system 108 preferably includes amotion sensor 114, a headset 116, and a controller 118. Motion sensor114 is configured to detect the location and orientation of headset 116,and transmit data accordingly. Controller 118 is configured to permit auser to interact with items in the simulation as viewed through headset116. Motion sensor 114 may be partially or fully integrated with headset116. Controller 118 may be physical or virtual, as part of headset 116.The details of a virtual reality system would be well understood, andfor simplicity, are not repeated herein.

Having described the preferred components of system 100, a preferredmethod of creating an indoor spatial mapping will now be described.Referring to FIGS. 1-3, an indoor spatial mapping is created by firststoring a two-dimensional floor plan 120 in floor plan database 104.Two-dimensional floor plans may be derived from multiple sources. Forexample, only, a 2-D floor plan may be digitally scanned with mappingdevice 102 on site, digitally scanned from a paper copy, providedelectronically from another source, or any combination thereof. Oncestored in plan database 104, the floor plan is formatted for 3-Dpresentation according to preferred steps set forth below.

Reference points are identified and digitally tagged. A reference pointmay be classified in according to fixed spatial, and moveable object(relative to other reference points). A tag for a fixed spatialreference point will include coordinates along the X, Y, and Z axes. Thecoordinates may be determined between reference points (localcoordinates based on distance and angular orientation), and/orgeographical coordinates. Local coordinates may be obtained by using adepth indicator on mapping device 102, manually measuring distances onsite, or through a local grid between devices such as Bluetooth beacons.Geographical coordinates may be obtained using an onboard GPS (ifavailable), in combination with accelerometers and/or gyroscopesoptionally integrated with mapping device 102 should a GPS signal not beavailable due to indoor interference.

A tag for a moveable object reference point may include coordinatesobtained in similar fashion as those for fixed spatial reference points.The tag for a moveable object reference point will additionally includean indicator that the reference point associated with the object ismoveable relative to one or more fixed spatial reference points. The tagwill further preferably include an indicator that the object isinteractable, and an object identifier indicating the nature of theobject, for example only, furniture or emergency equipment.

Objects that are interactive may be interacted with in one or more ways,such as moving, toggling, removing, rotating, lifting or setting down.An indicator indicating interactivity may be formed as a label and/or aspecial colour, highlighting or shading/shadowing.

A tag may be generated based on object recognition technology. Forexample only, image recognition technology may be used when scanning an3-D object with mapping device 102 if obtaining onsite dataelectronically. Image recognition technology may also be used whenscanning a 2-D floor plan into database 104 by recognising a symbol foran object. For example, a standard fire extinguisher symbol may berecognised as a fire extinguisher by processor 106, and the tag linkedto the object (fire extinguisher) may be configured so that processor106 renders a 3-D fire extinguisher based on the symbol shown in the 2-Dfloor plan.

Examples of items or areas tagged with fixed spatial reference pointsinclude permanent physical references such as entry and exits, doorways,windows, and stairs and/or lifts (if multi-level). Examples of itemstagged with moveable object reference points may include furniture, andemergency equipment such as a hose reel, fire blanket, and/or fireextinguisher 122. Examples of items tagged as interactive include a fireextinguisher, fire blanket, and/or hose reel. Thereafter, theinteractive tagged item is anchored to the two-dimensional floor plan inan initial, starting position (it being understood that the item may bemoveable within the simulation when rendered). The anchoring may beaccomplished by fixing the coordinates of an object or feature within ageospatial grid. Next, the floorplan, now enhanced with digital tags, issaved in database 104 as a latent 3-D floorplan and ready for renderingas an active, explorable 3-D floorplan with appropriate viewingequipment, as set forth further below. Periodic floor surveys may beused to modify and/or update floorplan data and change tag coordinatesand/or types as needed. Enhancing a digital plan with one or more tagsindicating interactability may be by way of colour, shadowing, orexplicitly with text and/or one or more symbols.

FIG. 5 shows a preferred method 200 of rendering the latent 3-Dfloorplan as an active, explorable 3-D spatial mapping. In step 202, theuser activates the mixed reality environment or system 108 (FIG. 1), forexample, sensor(s) 114, headset 116, and controller 118. Next, in step204, the user launches 1^(st) party mixed reality application, and, instep 206, connects to plan database 104 (FIG. 1). The plan database mayinclude more than just floorplans. For example, the plan database mayinclude a variety of profiles and data, such as a list of clients,campuses, buildings, and/or floors. In step 208, the user selects thefloor that they wish to view in 3-D, and the application downloads theselected floor's latent 3-D floorplan data. Next, in step 210, theapplication performs a 3-D extrusion, converting the tagged 2-D data toan explorable 3-D environment, compatible with mixed reality. In step212, the application loads and renders the 3-D explorable space 124(FIG. 4), displaying video output to mixed reality headset 116 (FIG. 1).Thereafter, in step 214, the user navigates through the 3-D environmentusing the mixed reality equipment, as shown in FIG. 4. As shown in FIG.4, within the explorable space, the user may interact with items taggedfor interaction, such as fire extinguisher 122 to put out a fire 126.Other examples may include a hose reel, a building alarm, an axe, etc.The interactable plan may include multiple layers of data to accommodatefixed object movement (e.g., a door that swings) and/or environmentalconditions such as smoke density.

It will be appreciated that the steps described above may be performedin a different order, varied, or one or more steps omitted entirelywithout departing from the scope of the present invention.

In order to help a user improve their training, the system may beconfigured to permit viewing by a trainer or supervisor on a monitorwhile the user is undergoing the 3-D simulation utilising the mixedreality system. The trainer may dynamically interact in the simulationduring the training exercise by creating emergency situations (e.g.,fires, etc.). At the end of the simulation, the user's performance maybe assessed and scored. A machine learning algorithm may be applied toidentify areas where the user could improve performance. Multipleperformances by a variety of users can be used by the machine learningalgorithm to indicate areas that could be improved in building safety(e.g., placement of emergency equipment, optimal size and placement ofbuilding features, etc.). Improved emergency preparedness by keypersonnel, coupled with improved building design after analysingevacuation patterns, will help save lives.

The foregoing description is by way of example only, and may be variedconsiderably without departing from the scope of the present invention.For example only, the mixed reality system may omit fixed sensors torely solely on motion sensors embedded in the headset. A controller,when used, may be configured as a wearable item or article of clothingsuch as a glove and/or shoe.

As an alternative to a virtual reality system with a headset, the mixedreality system may be configured on a smartphone to run augmentedreality technology.

Features of the invention may be adapted to external environments. Forexample, campus plans, walkways, courtyards, patios, balconies, parkinglots (or car parks) and areas between buildings may be converted from a2-D plan into a 3-D spatial mapping and simulation.

A variety of emergency scenarios may be simulated for evacuationplanning. For example only, artificial disasters such as a building fireor terrorist situation, or natural disasters such an earthquake,tornado, flood or hurricane may be simulated for practicing a buildingevacuation.

One or more of the embodiments described above may be configured foruses other than emergency simulation. For example only, creation of anindoor spatial map may be used for planning building extensions, orinstalling integrated building systems such as utilities, securityand/or fire alarm systems, and HVAC systems. The indoor spatial map mayalso be utilised as a base to optimally configure efficient use ofenergy (e.g., lighting, heating and air conditioning), or determine thebest placement of items such as emergency equipment. In one or moreembodiments, use in a gaming context (i.e., pitting players in liveaction against each other, particularly for entertainment) isspecifically excluded.

The features described with respect to one embodiment may be applied toother embodiments, or combined with or interchanged with the features ofother embodiments, as appropriate, without departing from the scope ofthe present invention.

The present invention in a preferred form provides the advantages of amore realistic environment and memorable simulation for buildingevacuations. By creating a simulation that is more memorable, keypersonnel, when faced with a real emergency, will act instinctivelybased on realistic practice rather than guessing and trying to rememberdetails of a floorplan while under pressure to leave. Includinginteractable environments and objects creates a more realisticsituation, and better prepares users for scenarios which may well beencountered. As has been well-researched, in an emergency situation,even saving a few seconds can save lives.

Other embodiments of the invention will be apparent to those skilled inthe art from consideration of the specification and practice of theinvention disclosed herein. It is intended that the specification andexamples be considered as exemplary only, with a true scope and spiritof the invention being indicated by the following claims.

What is claimed is:
 1. A system for indoor spatial mapping, comprising:a mobile mapping device having a camera; a plan database configured tostore 2-D floor plans; and a processor configured to integrate aselected floor plan from said plan database with object data receivedfrom said mobile mapping device to create an interactable 3-D spatialmapping of an interior portion of a structure so that one or moreobjects are moveable relative to a surrounding environment within the3-D spatial mapping, the object data including geospatial coordinates ofthe object, said processor being configured to digitally recognize saidmoveable objects from an image from said camera and spatially anchor therecognized moveable object in an initial position within theinteractable 3-D spatial mapping by fixing the coordinates of the objectwithin a geospatial grid.
 2. The system of claim 1, wherein the objectdata includes an item of emergency equipment.
 3. The system of claim 2,wherein the emergency equipment includes at least one of a fireextinguisher or a hose reel.
 4. The system of claim 1, wherein saidmobile mapping device is configured to tag predetermined objects foranchoring within the interactable 3-D spatial mapping.
 5. The system ofclaim 1, further comprising a virtual reality headset configured toreceive a data stream from said processor to render the 3-D spatialmapping.
 6. The system of claim 1, wherein said processor is configuredto deliver a data stream to a mobile communications device having acamera, the data stream causing the mobile communications device todisplay real-time surroundings in an augmented reality manner whenpositioned inside the structure.
 7. The system of claim 1, wherein saidprocessor is configured with image recognition technology.
 8. The systemof claim 1, wherein said processor is configured to anchor into the 2-Dfloor plan a 3-D object tag associated with a 2-D floor plan symbol. 9.The system of claim 1, wherein the interactable 3-D spatial mapping isconfigured to simulate smoke density.
 10. The system of claim 1, whereinthe recognized objects are tagged as either interactable or notinteractable.
 11. The system of claim 10, wherein recognized objectstagged as interactable include at least one of a fire extinguisher and ahose reel.
 12. The system of claim 1, wherein the interactable 3-Dspatial mapping is displayable only on a mobile communications device orvirtual reality headset.
 13. A method for generating an indoor spatialmapping, comprising: storing a 2-D floor plan in a floor plan database;digitally tagging reference points which includes at least one item ofemergency equipment in a level of a structure, the reference pointsincluding geospatial coordinates of the at least one item; anchoring theat least one tagged item of emergency equipment to the 2-D floor plan inan initial position by fixing the coordinates of the at least one itemwithin a geospatial grid; and rendering a 3-D floor plan with at leastone anchored item of emergency equipment, the anchored emergencyequipment being interactable within the 3-D floor plan, the at least oneitem of emergency equipment being tagged as an interactable 3-D objectbased on a recognized symbol contained in the 2-D floor plan.
 14. Themethod of claim 13, wherein the rendering of the 3-D floor plan includesdelivering the 3-D floor plan in an augmented reality format.
 15. Themethod of claim 13, wherein the rendering of the 3-D floor plan includesdelivering the 3-D floor plan in virtual reality format to a virtualreality headset.
 16. The method of claim 13, further including digitallyaltering the 3-D floor plan to incorporate a smoke density level. 17.The method of claim 13, wherein the at least one item of emergencyequipment includes a fire extinguisher.
 18. The method of claim 13,wherein at least one of the reference points is associated with anentryway, the reference point associated with the entryway being taggedwith local coordinate data.
 19. The method of claim 13, wherein at leastone of the reference points is associated with an entryway, thereference point associated with the entryway being tagged withgeographical coordinate data.
 20. The method of claim 13, wherein thereference points of the at least one item are tagged differently basedon whether the item is moveable.